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^ (54) Title: SYSTHM AND MH rilOD FOR CORRHCTINCJ MULTlPIJi AXIS DISPLACHMnN T DIS TORTION 

^ (57) Abstract: A .sy.sicm and method for correcting distortion in projected images caused by the projection of an input image by 
^ a projector lens onto a projection screen having a surface that is not orthogonal to the pmjcction axis of the projector. Projection 
^ parameters including the focal length of the projector lens and the angles that represent the pan, tilt and roll anulc of the projector 
are obtained. Then the projected area and the best viewable rectangular area that iit.s within said projected area is determined. The 
O distortion is then charactcri/cd by using a distortion transformation which corresponds to the iransformaiion that exists between Ihc 
^ vertices of the best viewable rectangular area and the corresponding vertices of the projected area. I-inally, the distonion Iransfor- 
»^ malion is inverted and applied to input image to obtain a distortion free pmjcclcd image. 
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Title: SYSTEM AND METHOD FOR CORRECTING MULTIPLE AXIS 
DISPLACEMENT DISTORTION 

FIELD OF THE INVENTION 

5 This invention relates to digital image processing and spatial 

image transformation, and in particular, to the correction of multiple axial 
displacement distortion in images. 

BACKGROUND OF THE INVENTION 

As is conventionally known, projecting an image onto a plane at 
10 an angle of elevation or depression causes keystone distortion. This is a 
common problem for projections where projectors are positioned below the 
screen to project upwards to the screen. The resulting images are generally 
known as having a "top-heavy" appearance. 

FIGS. 1 A, 1B and 2 illustrate the effect of off-axis projection. 

15 Specifically, FIG. 2 shows a projection configuration of a projector 5 and a 
screen 7 for the square image of FIG.1. The ideal projection shown in FIG. 
1A is from the ideal screen shown positioned along the y-axis and the top- 
heavy distorted projection shown in FIG. IB is from an actual screen 
positioned along a v-axis. As can be seen the projected image contains top- 

20 heavy distortion of the square image and that the non-uniform spacing grid is 
due to perspective effect as shown in FIG. 2. The size of the distortion 
depends on projection angle 6 , distance d from the projector 5 to the screen 
7 , and projector aperture 2a. FIG. 2 shows distortion along one axis only. 
Turning to FIG. 3. there is shown a projection diagram for the square image of 

25 FIG.1 distorted along two axis by the tilt and pan angles 0 . (p associated with 
projector 5 such that the distorted image shown on the projection screen (i.e. 
actual screen shown) results. 

One type of prior art approach to correcting distortion in 
projection systems is to use mechanical systems to move the optical axis in 
30 order to compensate for actual distortion. However, such mechanical prior art 
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system generally lack flexibility in that compensation is essentially limited to 
correcting one axis, usually along the vertical axis. 

Another prior art solution is (disclosed in U.S. 5,465,121 to 
Blalock et al. which uses a data processing system to correct distortion by 
5 removing pixels as necessary using a simple height and pixel equation. This 
system is, however, only effective for distortion along one axis, namely the 
vertical axis, and accordingly does not properly compensate for distortion 
along two or more axis. 

Another prior art solution is described in U.S. Patent 6,367,933 

10 to Chen et al., namely a method for correcting keystone distortion resulting 
from off-axis projection In two angles, horizontal, pan, and vertical, tilt. The 
method performs a sequence of two one-axis keystone distortions to achieve 
a two-dimensional conrection. First a correction for a tilt angle is corrected and 
then the resulting image is rotated by 90 degrees. The rotated image is 

15 corrected for a vertical angle distortion again but equivalent to the horizontal 
pan angle. The final image is obtained by rotating back the image by -90 
degrees. Rotation by +/- 90-degrees imposed intermediate memory to store 
rotated the whole or parts of rotated image. This approach introduces 
processing delays and memory cost. Further, this method is very specific to 

20 horizontal and vertical angles distortions only since rotation by +/- 90 degrees 
can be achieved without using very complex interpolation techniques. If a roll 
angle were applied as well such as that shown in FIG. 4 (or any other 
distortion like radial lens distortion are applied to the Image) then this method 
would require another specialized module to handle the situation, leading to 

25 processing delays and increased cost. 

It is therefore desirable to provide a general method and system 
to correct multiple axial displacement distortion in images, which addresses, 
in part, some of the shortcomings of providing undistorted images as 
discussed above. 
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SUMMARY OF THE INVENTION 

The invention provides in one aspect, a method for correcting distortion 
in projected images caused by the projection of an input image by a projector 
lens onto a projection screen having a surface that is not orthogonal to the 
5 projection axis of the projector, said projection screen being associated with 
assoiated with a display device, said method comprising the steps of: 

(a) receiving projection parameters including the focal length of the 
projector lens, the aspect ratio of the input image relative to the aspect 
ratio of the display device, and the angles that represent the pan, tilt 

10 and roll angle of the projector; 

(b) determining the projected area that results when distortion is 
present given the projection parameters from step (a) and then 
determining the vertices of a best viewable rectangular area that fits 
within said projected area such that the perimeter of said best viewable 

15 rectangular area is maximized; 

(c) characterizing the distortion caused by projection using a 
distortion transformation which corresponds to the transformation that 
exists between the vertices of the best viewable rectangular area and 
the corresponding vertices of the projected area; 

20 (d) inverting the distortion transformation to form a pre-distortion 

image transformation; and 

(e) applying the pre-distortion image transformation to the input 
image to obtain a distortion free projected image. 

A system for correcting distortion in projected images caused by the 
25 projection of an input image by a projector lens onto a projection screen 
having a surface that is not orthogonal to the projection axis of the projector 
said projection screen being associated with a display device, said system 
comprising: 

(a) a memory for storing projection parameters including the focal 
30 length of the projector lens, the aspect ratio of the input image relative 
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to the aspect ratio of the display device, and the angles that represent 
the pan, tilt and roll angle of the projector; 

(b) a processor coupled to said memory being adapted for 
determining the projected area that results when distortion is present 

5 given the projection parameters from step (a) and then determining the 

vertices of a best viewable rectangular area that fits within said 
projected area such that the perimeter of said best viewable 
rectangular area is maximized; 

(c) said processor also being adapted to characterize the distortion 
10 caused by projection using a distortion transformation which 

corresponds to the transformation that exists between the vertices of 
the best viewable rectangular area and the corresponding vertices of 
the projected area; 

(d) said processor also being adapted to invert the distortion 
15 transformation to form a pre-distortion image transformation; and 

(e) said processor also being adapted to apply the pre-distortion 
image transformation to the input image to obtain a distortion free 
projected image. 

Further aspects and advantages of the invention will appear 
20 from the following description taken together with the accompanying 
drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

In the accompanying drawings: 

FIGS. 1A and 1B are graphical representations of an ideal 
25 projection and a distorted projection of a square image on a projection screen; 

FIG. 2 is a schematic diagram of a conventional prior art 

projector; 

FIG. 3 is a schematic diagram of projection apparatus 
associated with the projected images of FIGS. 1A and IB; 
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FIG. 4 are graphical representations of the projected image of a 
square input image when the three axes of the projector are not aligned with 
the screen reference axes; 

FIG.5 is a block diagram of an example projection correction 
system of the present invention; 

FIG. 6 is a schematic diagram of a basic model of a projection 
system where the projection screen is plan; 

FIG. 7 is another schematic diagram of a model of a projection 
system where the projection screen Is not orthogonal to the projection axis; 

FIG. 8 is a schematic diagram illustrating the configuration of a 
input square Image projected onto the projection screen of FIG. 6 and the 
reference points used to compute the best viewable area; 

FIG. 9A is a schematic diagram of the projection screen on 
which is projected the square image of FIG.8; 

FIG. 9B is a schematic diagram of the rectangle of FIG. 8 
adjusted to match the aspect ratio of the display device; 

FIG. 1 0A is a schematic diagram of an output image having 
different aspect ratio than the display device; 

FIG. 1 0B is a schematic diagram illustrating the best viewable 
20 area for a projected 16:9 anamorphic image on a 4:3 display device; 

FIG. 1 0C is a schematic diagram illustrating the best viewable 
area for a projected 16:9 anamorphic image on a 4:3 display device optimized 
to the 16:9 aspect ratio; 

FIG. 11A is a schematic diagram illustrating the fonward 
25 transformation of a projected image where the image different aspect ratio 
than the display device; 

FIG. 1 1 B is a schematic diagram Illustrating the forward 
transformation of a projected image where the content has different aspect 



15 
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ratio than the display device and the projection area is optimize to match the 
content's aspect ratio. 

FIG. 11C is a schematic diagram illustrating the inverse 
transformation of a projected image where the content has different aspect 
5 ratio with display device. 

FIG. 1 1 D is a schematic diagram illustrating the inverse 
transformation of a projected image where the input image has different 
aspect ratio than the display device and the projection area is optimized to 
match the content's aspect ratio; 

10 FIG. 12 is a flowchart diagram showing the main process steps 

required to build the full range of projection distortion parameters according to 
one part of the correction method of the present invention; 

FIG. 13 is a flowchart diagram showing additional process steps 
required to build the polynomial interpolators for the range set of distortion 
15 parameters provided by the process of FIG. 12; 

FIG. 14A is a graphical representation of the distortion of an 
input square grid image when projected onto a projection screen; 

FIG. 14B is a graphical representation of the input image of FIG. 

14A; 

20 FIGS. 14C and 14D are graphical representations of the first 

and second passes of the pre-distorted input image of FIG. 14B which 
together result in pre-distorted image (FIG. 14D) which will compensate for 
distortion when projected; and 

FIG. 1 5 A is a schematic diagram illustrating the concatenation 
25 of projection and lens distortion; and 

FIG. 15B is a graphical representation of the pre-distorted input 
image that results from the concatenation illustrated in FIG. 15A. 
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DETAILED DES CRIPTION OF THE INVENTIQM 

FIG.5 illustrates an example projection correction system 10, 
built in accordance with the present invention. Specifically, correction system 
10 identifies various projection distortion parameters, generates an inverse 
5 transformation corresponding to the projection parameters, receives and pre- 
distorts input image data by applying the Inverse transfonnation. and outputs 
pre-distorted image data, which will appear distortion-free once projected on a 
screen (not shown). Specifically, correction system 10 includes a projection 
parameter module 12. a computational module 14. an image processor 16. a 
10 projection orientation detector 18. and a warp coefficient database 20. 

The projector orientation detector 18 may be any automatic 
device that captures the relative orientation between the projector and the 
projection screen. While it is preferred that the user manually inputs the 
orientation to the system and the focal length of the optical system, although it 

15 should be understood that projection orientation detector 18 could also be 
used to provide the angle of orientation. Three important pieces of data are 
provided to correction system 10. namely the angle of orientation of the 
projector (as will be discussed in detail in relation to FIG.7), the relative aspect 
ratio of the input image and the aspect ratio of the projector's display device. 

20 and the focal length f of the projector lens. The projection parameters are 
processed in order to find the proper index in a table of distortion parameters, 
which feed the image processor 16. 

Projection parameter module 12 receives a range of projection 
parameter data that relates to the relative angular position between the axis of 

25 projection of projector 5 and the surface of projector screen 7 for a variety of 
relative angular positions. The projection parameter data can be detected by 
projection orientation detector 18 and provided to projection parameter 
module 12. A range of projection parameter data is utilized for a manual 
solution and it should be understood that for an automated solution, correction 

30 system 1 0 provides the angular position in real time and projection parameter 
module 12 identifies the corresponding distortion coefficients. 
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Warp computation module 14 generates an image 
transformation which will compensate for multiple axis displacement 
distortion. Specifically, warp computation module 14 computes warp 
polynomials that correspond to the projection parameter data provided by 
5 projection parameter module 12. Projection parameter module 12 obtains 
appropriate polynomial warp coefficients from warp coefficient database 20. 
Each warp coefficient is a function of angular positions and is evaluated and 
formatted by warp computation module 14 to provide input data that is 
suitable for processing by image processor 16. 

10 Image processor 16, or warp engine, re-samples the input 

image according to the parameters of the inverse distortion. That is, the image 
transformation determined by warp computation module 14 is implemented in 
respect of the input image data so that when the inverse distorted input image 
is projected onto the screen, the final corrected image will result. As will be 

15 discussed, it is preferred for the image transformation to be performed as two 
ID image transformations for reasons of processing efficiency. As is 
conventionally, known, two ID transformations simply the hardware 
implementation for memory access and 2D filtering since a two dimension 
operation is reduced to two 1 D operations. 

20 The appearance of a projected image is affected by the specific 

angular position of projector 5 (i.e. the degree of rotation around three axes). 
Specifically, when there is off-axis displacement a distorted version of the 
input image will be projected on the screen. The distortion associated with the 
projected image is also function of the projector aperture or projector focal 

25 length, as well as lens geometry. The distortion can be completely defined by 
considering the 3D projection of a unit square where the center of projection is 
the optical center of projector, and where the intersection of projection cone 
with the screen surface is computed. 

In order to project a projected Image on as large an area as 
30 possible on projection screen 7, the projector field of view (FOV) is adjusted to 
it's maximum which is equivalent to choosing a minimum focal length. The 
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projected image is then a function of the projector field of view (FOV) as well 
as the angles between the screen global coordinates and the projector 
reference axis. Transformation of coordinates representing the displacements 
of the projector are R^ia) . for a rotation of angle a around an axis p , Thus. 

the transformation matrices written in the conventionally known notation of 
homogenous coordinates are as follows: 



Ryi(P) = 



10 0 0 

0 cosCv/-) -sin(v^) 0 

0 sin(v^) cos(va) 0 

0 0 0 1 

cos(<jt>) -sin(^) 0 0 

sin{q)) cos(<p) 0 0 
0 0 10 

0 0 0 1 



cos(0) 0 sin(0) 0" 

0 10 0 

-sin(0) 0 cos(0) 0 

0 0 0 1 



(1) 



where V is the pan angle of the projector measured 
10 counterclockwise from the vertical axis Y-axis, and 6 is the tilt angle of the 
projector measured positive in an upward direction around the horizontal X- 
axis and W the roll angle around Z-axis. 

Referring now to FIG. 6, there is shown a line diagram of a 
simple model of a projection system, where the zaxis is aligned with optical 
15 axis of the projector 5 and where the z axis is also normal to projection screen 
7. This is an ideal projection case, or ideal screen, where the projection q of 
an image point Pdoes not undergo any distortion. Only a scaling factor is 
applied to the projected image point P as a function of distance of the screen 
to the projector: 
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w-h f 
u-x — 

f 

w + / 

f 

W = )V 



(2) 



10 



15 



20 



As shown in FIG. 6, the focal length /Is the distance from the 
center of projection to the image plane, and s not equal to 0 is an arbitrary 
scaling factor introduced by homogenous coordinates. Also, w is the distance 
from the image plane to the screen plane. Homogenous coordinates allow for 
the representation of a non-linear operation by a linear matrix. In homogenous 
coordinates image point p and projected image point q are defined as follows: 



q= {u, V, w, 5)' , p = (x, y, z,l)' P = 



q=Pp 



0 
1 
0 

1// 



0 
0 

1 

0 



(3) 



where, again, /is the distance from the center of projection to 
the image plane. 

Referring now to FIG. 7, there is illustrated a line diagram of a 
model of a projection system showing projector 5 having a center of 
projection, and screen 7 that is rotated by certain angles ^ , 9 and 9 (i.e. 
geometric parameters) in the screen coordinates reference. It should be 
understood that the projector 5 is considered to be rotated and that the 
projection screen 7 is fixed. The projected point, ^(u,v,w) is displaced on the 
screen and follows the relation: 



q=PRyi(p)RAO)R,(¥)P 

p = R:\w)R:\e)R-\<p)p-' q 



(4) 
(5) 



BNSIXCIO: <WO 02101443A7 I > 



SUBSTITUTE SHEET (RULE 26) 



wo 02/101443 



11 



PCT/CA02/0O870 



As shown in FIG. 7, a square rectangle is projected onto a 
general quadrilateral. 

Referring now to FIG. 8, there is shown a screen diagram of a 
square projected onto a projected area Q on screen 7. The projected area Q 
5 is a general quadrilateral, but if screen 7 was perpendicular to the optical axis 
and the image plan parallel to the screen plane, the projected area would be 
square as well. As will be described, correction system 10 pre-processes the 
original image, using warp technology so that the projected image looks 
square on projection screen 7. 

10 The projection of a square, after correction, has to be a 

rectangle matching the display device aspect ratio. To optimize usage of the 
screen real state space, a largest rectangle, defined as Best Viewable Area 
(BVA) must be identified inside the quadrilateral. The BVA is the rectangle 
with the largest area, or with the highest spatial resolution inscribed onto the 

15 projected quadrilateral and with a desired aspect ratio. In the context of this 
embodiment, the BVA refers to the largest area. Since it is possible to adjust 
the input image aspect ratio, correction system 10 takes advantage of the 
elongated shape of the projected quadrilateral to find the largest image with 
desired aspect ratio that fits onto that elongated shape. For example, this 

20 approach gives a 16:9 wide screen image out of 4:3 aspect ratio display 
device. 

In order to simplify the computations it is assumed that the Best 
Viewable Area (BVA) with the largest area is approximated with the rectangle 
with the longest perimeter. Accordingly, if the vertices of the BVA are 
25 considered to be (^i,yi),(^2»J'i)»(^2»3^2) and(x,,j/2) , they can be determined by 
solving the equation (6) below: 
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maximize (Xj - ) + (3^2 " ) perimeter 
subject to : 

^2-^-^1 ordered vertices 

> y\ ordered vertices (6) 

^2 " 3^1 = P(^2 ~ ^] ) p : desired aspect ratio 

{x^.yj) inside g,for / = 1,2; 7 = 1,2 

Q is the quadrilateral with vertices (Uf^, Vj^J, ^ = 1, • - ,4 

Equation (6) is solved by linear optimization techniques like the 
simplex method as discussed in U.S. Patent No, 5,465.121 to Blalock et a!., 
v^hich is herein incorporated by reference. 

5 FIGS. 9A and 9B illustrate the Best Viewable Area (BVA) of 

FIG. 8 which is re-sized to make use of the largest available area on the 
screen. Using the zoom function on the projector 5 can be used to accomplish 
this step. FIG.9A illustrates the case where the input image and projector 5 
have the same aspect ratio; while FIG.9B shows the case where a wide 
10 screen 16:9 input image is rendered on a 4:3 display device (e.g. television 
screen). 

FIGS. 10A, 10B and 10C illustrate an application of the present 
approach to obtain an optimized Best Viewable Area (BVA) for a particular 
projected area Q. Correction system 10 takes advantage of the elongated 

15 shape distorted of the projected area Q and determines a BVA that 
corresponds to the wide screen aspect ratio. In this way, a cost effective 
projection of wide screen input video using inexpensive normal 4:3 display 
device can be achieved. When a 16:9 video is displayed on a 4:3 aspect ratio 
device, some display area is lost in order to avoid distortion in displayed 

20 image as shown in FIG, 10A. The projection of the image of FIG. 10A on off- 
axis screen using a 4:3 projector would result in a lost screen area inside a 
4:3 BVA as shown in FIG 10B. When the BVA is optimized to take into 
consideration the aspect ratio of the input image versus the display device 
aspect ratio, it can be seen on FIG IOC, that a bigger and better image is 

25 projected. 
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FIGS. 1 1 A to D shows the resulting warp grid that corresponds 
to the projection and aspect ratio conversion of FIG. 10. FIGS. 11 A and 11C 
show a input image projected at off-axis with a projector with the same aspect 
. ratio as input image. FIGS. 11A and 11C show a wide screen image (16:9) 
5 projected at off-axis with a 4:3 projector. It is clear in FIGS. 11C and 11D that 
the screen usage is better optimized when the aspect ratio conversion is 
introduced. 

FIG. 12 is a flowchart illustrating the main process steps 
associated with the correction method 50 of the present invention. At step 
10 (52), the projection parameters, namely focal length f, and a range for angles 
. e and cp relating to the angular rotation of screen 7. are obtained either 
from a projector orientation detector 18 as discussed above and/or the user. 
At step (54). a particular angular arrangement (i.e. one combination of angles 
V , 0 and <p) is chosen. 

15 At step (56), the warp polynomials that correspond to the 

projection parameters discussed above are determined. The transformation 
between the vertices of the Best Viewable Area (BVA) and the corresponding 
vertices of the projected quadrilateral Q defines the mapping required to 
characterize the distortion caused by projection on a non-ideal screen. This 

20 transformation is mathematically a perspective transformation and is defined 
by the following relation: 

u = U(x,y) - ^ 2_ 

(7) 

V = V(Xyy) = — — 2_ 

a^x + agj' + 1 

The eight unknown parameters ^^,/ = l . g of equation (7) are 
found by solving (7) by matching the vertices of the projected quadrilateral 
25 (w^Vj), (wj^Vj), (W3,V3) ,(W4,V4) to tho corresponding vertices of the BVA 

(^i.J'i). U2>>'i), ("2>^2).("i.v2) in FIG.8. Computational details can be located 
within U.S. Patent No. 5,465.121 to Blalock et aL which has been 
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incorporated by reference. Accordingly, the relations (7) define the distortion 
affecting an input image when it is projected. As conventionally known, image 
warping is a transformation that maps all positions from one image plane to 
corresponding positions In the second image plane. Equation (7) is a non- 
linear sampling of the input image and can be compensated by applying the 
inverse transformation of (7). before projection, namely: 



bjU -h 6gV 4- 1 



(8) 



The coefficients of the inverse transformation (8) are derived 
from their forward counterpart (7) by representing the transformations in 
matrix form and by computing the inverse matrix. In fact, equation (7) and (8) 
can be written In homogenous coordinates on the form: 









^b, 6j b^ ^ 


A = 


«5 «6 




b^ b^ b^ 




1 ^ 






(u 


V s) =A(^ 


y r)\ and (jc y 



For scaling factors s and r . The matrix B is found by inverting 
the matrix A using the adjoint matrix and the determinant of A matrix: 
B = adj{A)l^t\{A). 

In one embodiment described here and implemented in the 
svWI integrated circuit product manufactured by Silicon Optix Inc. of 
California and described in GEM Product Brief, Geometry Manipulator 
Reference Design, July 18, 2001. herein incorporated by reference, equation 
(8) is applied to the input image in two steps as a two-separable 
transformation. Separable transformations achieve better efficiency in term 
memory access and image re-sampling since the 2D processing is 
decomposed in a series of two ID re-sampling along orthogonal axis (rows 
and columns), this leads to simple hardware implementation. However, it 
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Should be understood that the relations of (8) could also be applied is a non- 
separable way as well. 

At step (58) the first pass of the decomposed 2D image 
processing transformation is performed. In the first pass, each vertical column 
is re-sampled according to the spatial transformation y(u,v) and an 
intermediate image /(«,>;) is generated: 



[„,v] [u,y] 



u,I{u,y)=-i i « 



b^u■\■b^v + \ 



(9) 



At step (60) the second pass of the decomposed 2D image 
10 processing transformation is performed. The intermediate image i{u,y) is 
expressed in term of spatial coordinates [u,y] where y is the final vertical 
position. But the transformation available EQ8, are functions of [u,v], not 
function of [u,y] . To find the re-sampling of /(„,;;). one needs to express 
X(u,v) in term of [u,y]. By variable substitution in equation (7), it can be 
15 shown that the separated expression is: 

C2 =b^bs-b:^b^ 
c^^b^-b^b^ 

c.^b^b.-b^b, ^'"^ 
C5 =b^bT-b^b^ 
Cg =b^ -bjj^ 

In the second pass, each horizontal line in is re-sampled 

according to the spatial transformation X\u,y) = X{u,Yiu,vy) 

lu,y\ -JM^ [x,y] (11) 

20 
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The embodiment described uses a polynomial approximation of 
equations (8) to (11). This kind of polynomial parameterization provides great 
flexibility and simplifies the task of composition with other type of distortions 
(e.g. Intrinsic projector distortion (lens distortion)). 

5 It should be noted that the first pass can be performed using the 

positional polynomial: 

10 Further, the second pass can be performed using the positional 

polynomial: 

jCYu^yJ = -.yu^c.y^c^u^a, ^ =±y,u^ (13) 

15 It should be noted that the approximation by polynomials (12) 

and (13) provides more flexibility since it allows for the concatenation of other 
spatial transformations which can be used to compensate for other image 
distortions (e.g. lens distortions). In contrast, the procedure for obtaining a 
parametric description of the distortion, equations (4) to (13) involves costly 

20 computations for a real-time embedded devices, like low cost projector. 

At step (56) the parameters of equations (12) and (13) for all 
values of the projection geometric parameters are computed. It should be 
understood that each coefficient A ^r^ in equations (12) and (13) of the 
position polynomial is a function of spatial positions in the image and angular 
25 parameters of the projection. For the example discussed here, angular 
positions are used for the sake of clarity. The polynomial coefficients from 
equations (12) and (13) are written as: 
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where equation (14) expresses surfaces in three dimensions 
that are approximated by polynomial approximation: 

0<k./,m<d 

^ - Where r,(x) is a polynomial of degree k in the variable x for a 

given polynomial basis. It is recommended to use B-splines and Chebychev. 
For the example discussed. T,(x) is a Chebyshev polynomial of the first kind 
of degree k. The example given in equation (15). is limited to three variables 
for the sake of clarity, but it should be understood that correction method 50 is 
1 0 applicable to any dimensions greater or less than three. 

At steps (58) and (60). the polynomials of (15) are stored in 
computer memory 64. The polynomials must be accessed and computed 
efficiently, so that low degree polynomial surfaces, <, 3. are fitted to the set 
of coefficients and y^^. These surfaces on the set of coefficients (14) of 
the transformation, provides an analytical description of the distortion and 
allow the warp to be performed at undefined angles by interpolation using 
equation (15). 

FIG. 13 is a flowchart diagram that illustrates the main process 
steps that follow from the method steps discussed in respect of FIG. 12. 
Specifically, the pre-computed surfaces (15) generated at steps (58) and (60) 
are used to dynamically find the distortion at a given parameter by 
interpolation. This process also provides for efficient geometric data 
compression since the memory required to represent the set of all possible 
distortions is reduced, which is a significant advantage for embedded 
implementation where memory and computation resources are often very 
tightly constrained. 



15 



20 



25 



SUBSTITUTE SHEET (RULE 26) 



wo 02/101443 



18 



PCT/CA02/00870 



At Step (102). the coefficient sets are retrieved from memory 64. 
At step (104), the coefficient space is partitioned into a list of elementary 
surfaces. At step (106), the two most similar surfaces are merged and then at 
step (108), it is determined whether there has been a fit. If so then at step 
5 (109) S is added to the list of elementary surfaces. If not, then at step (110) it 
is determined whether any more merge is possible and if so, step (106) is 
repeated. If not, then at step (112) the fitting degree is incremented. At step 
(1 14), it is determined whether the degree of the fitting polynomial is less than 
the maximal degree imposed by the user. If so, then step (106) is repeated. If 
10 not, then the polynomial coefficients surface are saved in memory 118. 

FIG. 14A is a graphical representation of the distortion of an 
input square grid Image when projected onto a projection screen showing the 
distortion that results from axial displacement. FIG. 14B is a graphical 
representation of the input image of FIG. 14A. FIGS. 14C and 14D are 
15 graphical representations of the first and second passes of the pre-distorted 
input image of FIG. 14B to compensate for distortion when projected, where 
the resulting pre-distorted image shown in FIG. 14B, when projected, will 
result in a non-distorted projected image. 

In a real projection situation, geometric distortion descriptions 
20 are extrinsic to the projector 5. since it depends more on positional (rotation 
angles, distance) information. For the most accurate distortion correction, 
intrinsic parameters of the projection system need to be considered, mostly 
lens distortion (radial and tangential). It is clear that the method developed in 
equations (15) is general enough to handle such a situation. That is, the 
25 method described for obtaining the warp polynomial is general enough to 
allow those skilled in the art to include in the distortion other parameters (e.g. 
barrel or pincushion lens distortion). A concatenation of distortions can be 
achieved by applying equation (6) to the image as distorted by intrinsic 
projector parameters as shown in FIGS. ISA and 158. The remaining part of 
30 the development remains the same. The intrinsic distortion of the projector 
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can either provided by the manufacturer or can be determined during a 
calibration process. 

Accordingly, correction system 10 and correction method 50 and 
100 rectify deformations in projected images caused by general projection of 
5 a source .mage onto a screen, when the surface of the screen is not 
perpendicular to the symmetry axis of the projector lens. Specifically 
geometric parameters are provided to a warp engine to correct the 
complicated distortions encountered in projection. To obtain a distortion free 
projected image, the source image is pre-distorted in the opposite way to 
10 compensate for actual distortion by a warp engine. 

it should be understood that while this invention has been 
described in terms of projected images, projectors, and projection systems 
th,s invention is equally applicable to transforming images taken at one 
angle/location to images taken from another angle/location. Also while 
distortion has been described in the projection case, the same issues arise for 
an image as seen by camera or recorder where the camera or recorder is off- 
axis from the image. Thus, an image taken one angle has to be distorted or 
warped so as to appear to be taken from another angle. 

As will be apparent to those skilled in the art various 
modifications and adaptations of the structure described above are possible 
without departing from the present invention, the scope of which is defined in 
the appended claims. 
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Claims 

1 A method for correcting distortion in projected images caused by 

the projection of an input image by a projector lens onto a projection screen 
having a surface that is not orthogonal to the projection axis of the projector, 
said projection screen being associated with assoiated with a display device, 
said method comprising the steps of: 

(a) receiving projection parameters including the focal length of the 
projector lens, the aspect ratio of the input image relative to the aspect 
ratio of the display device, and the angles that represent the pan, tilt 
and roll angle of the projector; 

(b) determining the projected area that results when distortion is 
present given the projection parameters from step (a) and then 
determining the vertices of a best viewable rectangular area that fits 
within said projected area such that the perimeter of said best viewable 
rectangular area is maximized; 

(c) characterizing the distortion caused by projection using a 
distortion transformation which corresponds to the transformation that 
exists between the vertices of the best viewable rectangular area and 
the corresponding vertices of the projected area; 

(d) Inverting the distortion transformation to form a pre-distortion 
image transformation; and 

(e) applying the pre-distortion image transformation to the input 
image to obtain a distortion free projected image. 

2. The method of claim 1, wherein step (e) comprises applying the pre- 
distorted image transform as a 2D image transform. 

3. The method of claim 1, wherein step (e) comprises applying the pre- 
distorted image transform as two separated 1D image transforms. 

4. The method of claim 3, wherein step (e) comprises the steps of: 
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(i) 

(ii) 
(iii) 



separating the pre-distortion image transformation into a first 
pass transformation and a second pass transformation; 
applying the first pass transformation using a first positional 
polynomial; and 

applying the second pass transformation using a second 
positional polynomial. 



5 



The method of claim 1. wherein step (a) comprises receiving 
projection parameters for a range of angle sets, wherein each angle set 
contains angles that represent the pan. tilt and roll angle of the projector lens 
wherein steps (b) to (d) are calculated for each angle set such that a pre- 
distortlon image transform for a particular angle set can be dynamically 
obtained and applied to the input Image. 

6. The method of claim 1, wherein said method achieves correction of 
another distortion by executing step (b) such that the distorted image is the 
projected image that results from inclusion of said another distortion. 

7. A system for correcting distortion in projected images caused by the 
projection of an input image by a projector lens onto a projection screen 
having a surface that Is not orthogonal to the projection axis of the projector 
said projection screen being associated with a display device, said system 
comprising: 

(a) a memory for storing projection parameters Including the focal 
length of the projector lens, the aspect ratio of the input image relative 
to the aspect ratio of the display device, and the angles that represent 
the pan. tilt and roll angle of the projector; 

(b) a processor coupled to said memory being adapted for 
determining the projected area that results when distortion is present 
given the projection parameters from step (a) and then determining the 
vertices of a best viewable rectangular area that fits within said 
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projected area such that the perimeter of said best viewable 
rectangular area is maximized; 

(c) said processor also being adapted to characterize the distortion 
caused by projection using a distortion transformation which 
corresponds to the transformation that exists between the vertices of 
the best viewable rectangular area and the corresponding vertices of 
the projected area; 

(d) said processor also being adapted to invert the distortion 
transformation to form a pre-distortion image transformation; and 

(e) said processor also being adapted to apply the pre-distortion 
image transformation to the input image to obtain a distortion free 
projected image. 

8. The system of claim 7, wherein said processor is also adapted for 
applying the pre-distorted image transform as a 2D image transform. 

9. The system of claim 7, wherein said processor is also adapted applying 
the pre-distorted image transform as two separated 1D image transforms. 

10. The system of claim 9, wherein 

(i) said processor is also adapted to separate the pre-distortion 
image transformation into a first pass transformation and a 
second pass transformation; 

(ii) said processor is also adapted to apply the first pass 
transformation using a first positional polynomial; and 

(iii) said processor is also adapted to apply the second pass 
transformation using a second positional polynomial. 

1 1 . The system of claim 7, wherein said processor is adapted to receive 
projection parameters for a range of angle sets, wherein each angle set 
contains angles that represent the pan, tilt and roll angle of the projector lens, 
and wherein said processor is adapted to calculate a pre-distortion image 
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transform for each angle set such that a pre-distortion image transform for a 
particular angle set can be dynamically obtained and applied to the input 
image. 



12. The system of claim 7. wherein said processor achieves correction of 
another distortion by determining the projected area that results when 
distortion and said another distortion is present given the projection 
parameters and then determining the vertices of a best viewable rectangular 
area that fits within said projected area such that the perimeter of said best 
viewable rectangular area is maximized. 
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